A Method for Determining the Maximum Number of Nonoverlapping Linear Test Forms That Can Be Assembled From an Item Pool (CT 03-04) (PDF)

A valuable resource for any testing agency is its item pool. This paper considers the problem of assembling the maximum number of nonoverlapping linear test forms from an item pool. The forms are assembled by solving a sequence of integer programming problems. If specified conditions are satisfied, the procedure assures the assembly of the maximum number of linear forms supported by the pool. If the conditions are not satisfied, multiple linear forms are assembled and an upper bound and a lower bound on the maximum number of forms are provided. All integer programming problems are solved with commercially available software. Introduction A valuable resource for any testing agency is its item pool. Knowing the maximum number of linear test forms supported by an item pool assists with monitoring the pool and guides the development of new items. The availability of several nonoverlapping forms provides flexibility in test form presentation and alleviates item exposure concerns. The statistics associated with items make certain items likely candidates to be used on a test form, and other items not so likely to be used. Determining the maximum number of nonoverlapping forms will promote the spread of “good” items across all forms. The last twenty years have seen a widespread usage of automated test assembly at testing agencies. Most practical test assembly problems are NP-Hard (Garey & Johnson, 1979). This implies that no polynomial algorithm exists for their solution and a search procedure must be used for large problems. An exception is when the test assembly problem can be expressed as a network flow problem (Ahuja, Magananti & Orlin, 1996). A network flow problem can be solved in strongly polynomial time; that is, an upper limit on the number of iterations to solve the problem can be expressed as a polynomial in the number of nodes and arcs in the network. An example of a test assembly problem that could be modeled as a network flow problem is one where the only constraints are a distribution of the item difficulty and a hierarchical cognitive content requirement. For this special case, it would be possible to solve a single problem that would assemble any number of test forms from the item pool. The number of test forms being assembled could be increased gradually and, when no feasible solution exists, the maximum number of forms supported by the pool would have been found. This paper considers the more commonly encountered situation where the assembly problem cannot be expressed as a network flow problem. In this case, the assembly of the maximum number of nonoverlapping test forms from an item pool has not been (to our knowledge) successfully solved. While most test assembly problems are NP-Hard, this does not mean that the assembly of a single linear test form is difficult. Most test assembly situations do not require the optimization of an objective function. Test specifications are defined and any combination of items meeting the specifications yields an acceptable test form. A typical item pool would give rise to a large number of ways to combine items to make a form. The more notable heuristic methods for test assembly are Luecht and Hirsch (1992), Luecht (1998), and Swanson and Stocking (1993). Armstrong, Jones, and Kunce (1998) and Armstrong, Jones, and Wu (1992) utilize network flow and Lagrangian relaxation for test assembly. Theunissen (1985), van der Linden (1998), and van der Linden (2000) propose the use of a more general mixed integer programming (Nemhauser & 1